Papers with loss selection
Pre-training Distillation for Large Language Models: A Design Space Exploration (2025.acl-long)
Copied to clipboard
| Challenge: | Knowledge distillation (KD) aims to transfer knowledge from a large teacher model to a smaller student model for model compression. |
| Approach: | They extend knowledge distillation to the pre-training phase of large language models . they first conduct an experiment using a teacher LLM to distill a 1.9B student LLM . |
| Outcome: | The proposed model can be used to distill a 1.9B student model using a teacher LLM. |